To generate a three-dimensional image from a two-dimensional image, there is a method of using a depth signal which is generated by estimating a depth structure of the two-dimensional image. Three-dimensional image generating apparatus using such a method includes a depth signal generating apparatus. For example, the three-dimensional image generating apparatus generates a two-dimensional image of a viewpoint, which is different from a viewpoint at which the input two-dimensional image is obtained, by using a generated depth signal, and generates a three-dimensional image signal including a multi-viewpoint two-dimensional image. A conventional depth signal generating apparatus combines a plurality of depth signals and outputs a combined depth signal. However, when depth signals are combined, contrast of the depth signal deteriorates although rapid change of depth with a lapse of time does not easily occur. As a result, the depth feel and stereoscopic effect of the three-dimensional image displayed on a display device are deteriorated.